Feature Selection For Gene Selection And Prediction

In many machine learning applications, one must perform feature selection in order to obtain good classification performance. For example, selecting a good feature subset is critical when the sample size is small compared with the dimesionality and noise in the observations. When this is the case, it is necessary to reduce the number of features to avoid modeling noise in the classifier. When the number of features becomes large compared with the class separation, the curse of dimensionality sets in. Many possible decision boundaries are possible that give zero error over the training sets, but do not generalize well [3]. By contrast, using too few features or a poor subset may not allow for the classes to be well-separated in feature space. Bayesian variable selection using Gibbs sampling works by maximizing the posterior likelihood of a classifier model numerically. This is performed by simulating draws from the likelihood density function using a Gibbs sampler. The Gibbs sampler is a technique for drawing from a probability density function (PDF), when the actual form of the PDF may not be possible to compute analytically [2]. Several issues present themselves in feature selection, of which one of the most important is the selection of the number of variables. For a given classification problem with a set number of training examples, the error of the classifier decreases as features are added until it reaches a trough, and then increases again, as shown in figure 2 [3], [8]. For this reason one wishes to have a means to automatically determine the optimal number of variables to use for classification. Another related issue is the ranking of features. This is useful to set the number of features by hand by ordering features according to usefulness, and using the ranking to set a cutoff. Last, features may be useful in conjunction with each other, or may be redundant. In designing a classifier, one wishes to get rid of the redundant features, while exploiting relationships between features that may not yield good classification performance when taken on their own.